Probabilistic Exploration in Planning while Learning

Sequential decision tasks with incomplete infor­ mation are characterized by the exploration prob­ lem; namely the trade-off between further exploration for learning more about the environ­ ment and immediate exploitation of the accrued information for decision-making. Within artificial intelligence, there has been an increasing interest in studying planning-while-learning algorithms for these decision tasks. In this paper we focus on the exploration problem in reinforcement learn­ ing and Q-learning in particular. The existing exploration strategies for Q-learning are of a heu­ ristic nature and they exhibit limited scaleability in tasks with large (or infinite) state and action spaces. Efficient experimentation is needed for resolving uncertainties when possible plans are compared (i.e. exploration). The experimenta­ tion should be sufficient for selecting with statis­ tical significance a locally optimal plan (i.e. exploitation). For this purpose, we develop a probabilistic hill-climbing algorithm that uses a statistical selection procedure to decide how much exploration is needed for selecting a plan which is, with arbitrarily high probability, arbi­ trarily close to a locally optimal one. Due to its generality the algorithm can be employed for the exploration strategy of robust Q-learning. An experiment on a relatively complex control task shows that the proposed exploration strategy per­ forms better than a typical exploration strategy. continuous flow of events in time. Effective decision-mak­ ing requires resolution of uncertainty as early as possible. The . te?dency to minimize losses resulting from wrong predictions of future events necessitates the division of the problem solution into steps. A decision at each step must make use of the information from the evolution of the events experienced thus far, but that evolution, in fact, depends on the type of decision made at each step.